Skip to content

feat: add SQL_QUERY dashboard filter type to the schema - #3672

Open
karl-power wants to merge 4 commits into
mainfrom
sql-dashboard-variables
Open

karl-power wants to merge 4 commits into
mainfrom
sql-dashboard-variables

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

Adds the SQL_QUERY dashboard filter type to the schema: a variable-only filter whose dropdown values come from the first column of a free-form ClickHouse query. This is the first of four PRs; it adds the data model and validation only, so the filter can't be created or used in the UI yet.

  • SqlQueryDashboardFilterSchema stores a connection ID and a sqlTemplate, with isBroadcastEnabled: false and isVariableEnabled: true fixed, since there's no column to broadcast.
  • isSqlQueryFilter type guard.
  • validateDashboardFilterQueries expands the template with empty variable selections and flags $__sourceTable / $__filters (the filter has no source) and any macro or variable expansion error. The SQL itself isn't parsed; ClickHouse reports syntax errors when it runs.
  • Dashboard template export replaces the filter's connection ID with the connection name, as it does for raw-SQL tiles.
  • The external API and MCP treat the type as internal-only, like ADHOC, until they expose it.
  • The filter settings list handles the new type in its exhaustive switch, labelling it "SQL query". It's only visible for a filter saved through the API or a JSON import, since the UI can't create one yet.

How to test on Vercel preview

N/A — non-UI change

References

  • Linear Issue: HDX-5524
  • Related PRs: follow-ups will add API connection validation and MCP / external API support, the dropdown values lookup, and the filter editor form.

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 10:56
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 3:54pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 3:54pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 369bd81

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 6
  • Production lines changed: 248 (+ 202 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@karl-power
karl-power force-pushed the sql-dashboard-variables branch from 989bc97 to f94e9ab Compare November 16, 2026 10:57
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 466 passed • 1 skipped • 1035s

Status Count
✅ Passed 466
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/common-utils/src/core/utils.ts Resolved
Comment thread packages/api/src/utils/externalApi.ts Resolved
Comment thread packages/common-utils/src/filters.ts Resolved
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 3 minor

5 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Schema-only PR adding the SQL_QUERY dashboard filter type (zod discriminated union member, template-macro validation, dashboard-template export, and import-page connection mapping). No runtime values lookup or editor yet — this is the first of four stacked PRs. Reviewed against base 743c3c1a.

✅ No critical issues found. No P0/P1 ship-blockers: nothing introduces a guaranteed happy-path crash, data loss on a normal path, or an injection vector. The new union member, type guard, Extract/Exclude external-compat type, and satisfies never exhaustive checks are all sound. The recommendations below are edge-gated correctness risks and coverage gaps.

🟡 P2 -- recommended

  • packages/common-utils/src/core/utils.ts:957 -- Exporting a SQL_QUERY filter whose connection id no longer resolves writes connection: '', but SqlQueryDashboardFilterSchema.connection is z.string().min(1), so re-importing the template fails DashboardTemplateSchema parse and the entire import is rejected, not just that filter.
    • Fix: Relax the exported connection to tolerate an empty sentinel (as the raw-SQL tile's z.string() does) so an unresolved connection becomes a mappable import row instead of a hard schema rejection.
    • adversarial, api-contract, correctness, testing
  • packages/app/src/components/DashboardFiltersModal/filterFormState.ts:107 -- An imported SQL_QUERY filter renders an Edit button (DashboardFiltersList.tsx:184), but toFormValues/toSavedFilter have no SQL_QUERY case, so saving falls through to the generic DashboardFilterSchema.parse and throws on the missing connection/sqlTemplate.
    • Fix: Hide or disable the Edit affordance for SQL_QUERY filters until the editor supports them, or carry connection/sqlTemplate through the form state.
    • correctness
  • packages/api/src/utils/externalApi.ts:255 -- The new !isSqlQueryFilter(filter) exclusion branch in getExternalCompatibleFilters has no test, so a regression that re-exposes SQL_QUERY through the external API or MCP would go uncaught.
    • Fix: Add a unit test asserting getExternalCompatibleFilters drops both ADHOC and SQL_QUERY while retaining the other filter types.
    • testing, kieran-typescript
  • packages/app/src/DBDashboardImportPage.tsx:945 -- The new SQL_QUERY filter branch of the connection-propagation effect — the core behavior the effect was rewritten for — is untested; existing import tests only cover auto-map-on-mount and block-on-unmapped.
    • Fix: Add an interaction test that changes one SQL_QUERY filter's connection and asserts other filters/tiles sharing the input connection name are remapped while already-dirty rows are left untouched.
    • testing, maintainability, correctness, adversarial, julik-frontend-races
🔵 P3 nitpicks (4)
  • packages/app/src/DBDashboardImportPage.tsx:1137 -- connection!.id for a SQL_QUERY filter can throw a TypeError (surfacing only as a generic "Something went wrong" toast) if the mapped id becomes unresolvable between validation and submit; superRefine only checks the mapping string is non-empty, not that it resolves.
    • Fix: Guard the lookup (if (!connection) return filter;) or verify the id resolves in superRefine; mirrors the existing raw-SQL tile pattern at line 1100.
  • packages/api/src/utils/externalApi.ts:265 -- getExternalCompatibleFilterValues still strips only ADHOC values, not SQL_QUERY, diverging from the internal-only intent; harmless today only because no values lookup exists yet.
    • Fix: Extend the strip to SQL_QUERY values when the follow-up adds the values lookup.
  • packages/common-utils/src/filters.ts:862 -- validateSqlQueryFilterTemplate surfaces an arbitrary caught e.message directly into the import warning UI; a non-macro error would leak an internal message to the user.
    • Fix: Map unexpected (non-macro) errors to a generic message and log the detail instead of rendering it.
  • packages/app/src/DBDashboardImportPage.tsx:921 -- The rewritten connection-propagation effect carries two near-identical change-detection blocks and a shared mutable inputConnectionName/connectionId sentinel, raising the cost to reason about it (behavior verified correct).
    • Fix: Extract a detectDirtyChange(mappings, prevRef, fieldPrefix) helper for the tile and filter branches and union the resulting key lists, keeping the single effect.

Pre-existing (not counted toward verdict)

  • packages/api/src/routers/external-api/v2/dashboards.ts:3633 -- The external/MCP full-update path replaces the filters array via convertExternalFiltersToInternal, so a GET→modify→PUT round-trip silently drops internal-only filters the client never received; this diff extends that existing ADHOC data-loss hazard to SQL_QUERY. Worth addressing in the follow-up that exposes SQL_QUERY (merge-preserve internal-only filters on update).
    • api-contract, agent-native

Prior comments

All four substantive prior threads were verified addressed in the current diff: connection mapping + block-until-mapped (93a610bd9), per-filter detail warning line (96a5282c5), and the two explicitly deferred follow-up items (externalApi.ts hidden filters, DashboardFiltersList.tsx display). The jscpd duplicate-code advisory (core/utils.ts ~988-1008 ↔ 1036-1056) is pre-existing boilerplate shared by two distinct converters, only re-flagged because the added connections argument shifted line numbers — not worth deduping here.


Reviewers (11): correctness, adversarial, kieran-typescript, api-contract, julik-frontend-races, testing, maintainability, project-standards, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • The combined-macro rejection message ($__sourceTable and $__filters both present, joined by and) in validateSqlQueryFilterTemplate is not exercised — only single-macro cases are.
  • convertToFilterTemplate's SQL_QUERY unresolved-connection ?? '' fallback is untested (only the raw-SQL tile fallback is covered).
  • SqlQueryDashboardFilterSchema's fixed literals are tested for wrong values but not the undefined/unset case, unlike the static-list variant.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions github-actions Bot added review/tier-3 Standard — full human review required and removed review/tier-2 Low risk — AI review + quick human skim labels Nov 16, 2026
Comment thread packages/common-utils/src/core/utils.ts Resolved
Comment thread packages/api/src/utils/externalApi.ts Resolved
Add a variable-only dashboard filter whose dropdown values come from the
first column of a ClickHouse query run on a connection.

- Add SqlQueryDashboardFilterSchema and the isSqlQueryFilter guard
- Validate the template's macros and variable references, rejecting
  $__sourceTable and $__filters
- Export the filter's connection as a name in dashboard templates
- Keep the type out of the external API and MCP for now

Refs HDX-5524
@karl-power
karl-power force-pushed the sql-dashboard-variables branch from dde72dc to 369bd81 Compare November 16, 2026 15:50
tooltip: 'Sources the condition keys and values come from',
label: getSourceNames(filter.sources, sources),
};
case 'SQL_QUERY':

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — An imported SQL_QUERY filter can be edited, but the edit form drops its connection and SQL and can't save it

Imports can now create SQL_QUERY filters, and the filter list shows them with the usual edit pencil. Clicking it runs toFormValues (filterFormState.ts:60), which has no connection or sqlTemplate fields, so both values are lost. DashboardFilterEditForm.tsx:259 then falls through to QueryExpressionFilterEditForm, and the type picker shows nothing because FILTER_TYPE_OPTIONS has no SQL_QUERY entry. Saving hits the generic branch of toSavedFilter, and DashboardFilterSchema.parse throws ('This filter is incomplete…'). If the user switches the type instead, the SQL template is silently discarded. Until the editor PR lands, hide or disable the edit action for SQL_QUERY filters, or keep their fields in the form state and save them unchanged.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

return (filters ?? []).filter(
(filter): filter is ExternalCompatibleDashboardFilter =>
!isAdhocFilter(filter),
!isAdhocFilter(filter) && !isSqlQueryFilter(filter),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — MCP and external-API saves silently delete SQL_QUERY filters, breaking tiles that use their variable

These reads now hide SQL_QUERY filters, but the writes replace the whole filter array: MCP saveDashboard.ts:386-397 only re-adds hidden PromQL filters, and external PUT v2/dashboards.ts:3632 writes filters directly. So a GET→modify→save round trip removes every SQL_QUERY filter, and raw-SQL tiles referencing its $var lose their variable. Before hiding the type, carry existing internal-only filters (SQL_QUERY, and ADHOC too) through as hiddenFilters, the way PromQL filters are handled.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

{warning.where}
</Text>
{warning.detail && (
<Text c="yellow" size="sm">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New warning detail text uses raw palette color c="yellow" instead of the semantic Text variant

The conventions say to use <Text variant="warning"> for status text rather than raw Mantine colors, and to migrate nearby call sites when you touch them. Use variant="warning" on the new detail Text and the reworked line above it (line 342).

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* Why a SQL query filter's template can't expand, or undefined when it can.
* The SQL itself is not parsed: ClickHouse syntax is only checked when it runs.
*/
export function validateSqlQueryFilterTemplate(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — validateSqlQueryFilterTemplate is exported but nothing outside this file calls it

Its only caller is validateDashboardFilterQueries in the same module; grep finds no import in app, api or tests. Keep it module-private until the follow-up PR that needs it, or add a direct unit test so the export has a reason to exist.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

]);

// Propagate connection mapping changes to other RawSQL tiles with the same input connection
// Propagate connection mapping changes to other RawSQL tiles and SQL query

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The rewritten connection-propagation effect has no test

This effect now propagates connection picks in both directions, tile→filter and filter→tile, using two separate prev-refs. The new tests only cover auto-mapping by name and the unmapped-connection error. Add a test with a raw-SQL tile and a SQL_QUERY filter that share a template connection: pick a connection on one row and assert the other row follows, and that a row the user already changed (dirty) is left alone.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 369bd81d Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — 369bd81d Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants